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ABSTRACT 



The purpose of this thesis is to provide information about 
wargaming to prospective users. An in depth discussion focusing on 
two primary components of wargaming is presented. The first 
component is wargaming in general. A description is provided of 
wargaming features and a discussion concerning the limitations and 
hazards inherent to wargames. The second component is how to 
design a wargame experiment to support command, control and 
communications (C3) hypotheses testing. A description of the steps 
in planning a general wargame experiment including detailed 
information on wargame software considerations is presented. 
Finally a synopsis followed by a critique of an actual C3 wargame 
experiment is provided as an example of the principles given in 
this thesis. 
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I. INTRODUCTION 



A. BACKGROUND 

Peacetime operations that can be observed stimulate little 
of the activity that can be expected in crises or wartime, and 
fleet exercises that offer an opportunity to observe combat 
direction offer little opportunity to observe higher level 
command functions in a realistically stimulated environment 
(Ketron, Inc. 1979, p. 4). Before Desert Storm, many weapons, 
systems, and ideas were untested in combat. The Navy cannot 
rely on actual conflict for evaluating its combat 
capabilities. Wargaming used as a tool in research, 
operational planning, and training is rapidly expanding to 
meet these needs. 

Improvements will be needed to adapt existing systems and 
technology and to make limited upgrades and changes in a 
number of areas, including, the following :... Simulation- 
gaming support to assist in near real-time analysis of 
alternative courses of action and for peacetime training 
and readiness.... (Command and Control Functional 
Analysis and Consolidation Review Panel Report, 1991, p. 

7) 

Interest in improving the Navy command, control and 
communication (C3) systems has been encouraged by the 
Department of Defense in recent years for several reasons. 
The threat to Naval forces continues to become more 
complicated due to complex, multi-dimensional anti-ship 
missiles deployed on aircraft, ships, and submarines, and 
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increasingly sophisticated electronic warfare technology. 
There has been an exponential growth in the amount of 
information available to commanders for assessing the 
situation and operational planning. Rapid advances in 
computer and communications technologies have enabled the 
planning and design of increasingly sophisticated systems to 
support tactical C3. 

Modeling, simulation and gaming can effectively be used 
for identifying requirements and designing, developing, 
acquiring, training for and employing major C3 systems. 
Command and control of military forces and assets is 
ultimately a human decision process supported by C3 systems 
with human operators receiving imperfect information. The 
most effective and economical way to evaluate systems 
involving human processes is by using wargaming. In 
particular, wargaming best supports research for the following 
command and control functions: situation monitoring, decision 
making, and management of forces and intelligence support 
assets . 

The first step in command and control research (C2R) 
focuses on decisions critical to military operations, the 
two most important being to identify decisions that have 
an impact on combat outcome; and to identify the 
appropriate command levels where different decisions are 
most appropriate. Once the crucial decisions are 
identified, C2R then encompasses many issues, including 
identifying information required to make the best 
decisions and the effect of time on the information 
gathering and dissemination process. (Hogan and Might, 
1988, p. 33) 
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Navy officers need to be aware of the benefits and hazards 

of using wargaming for research, operational planning and 

training. Decision makers, who have not been involved in 

wargaming, can expect to deal with games or the results of 

games in the course of their careers. Wargame use is as 

complex as the situations they attempt to address. 

The fact is that wargames and wargaming are consistently 
misunderstood, denigrated, even denounced, not only by 
gaming outsiders, but also by gaming proponents and 
practitioners. This unfortunate situation is a result of 
a failure to reconcile the fundamental ambiguities of 
wargaming, to understand the nature of the tool - the game 
- and of the process of using it - gaming. (Perla, 1990, 
P-2) 

In addition to limitations inherent in the games, game 
data users need to question and understand the process of 
applying a wargame to an experiment. An example of wargame 
abuse is the wargame that was conducted in the spring of 1942 
at the War College in Tokyo. Japanese naval planners were 
preparing plans for the capture of Midway and the western 
Aleutians in early June, the seizure of strategic points in 
New Caledonia and the Fiji Islands in July, air strikes on 
southeast Australia, and operations against Johnston Island 
and Hawaii in August. 

During the play the Nagumo Force was attacked by land- 
based air while its own planes were attacking Midway. 
Following the rules of the game, an umpire determined 
that the carriers received nine hits and that two of 
them, the Akagi and Kaga, were sunk. Rear Admiral 
Ugaki, the director of the game, arbitrarily reduced 
the number of hits to three, and the number of 
sinkings to one, and then permitted the sunken carrier 
to participate in the next part of the play dealing 
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with the New Caledonia and Fiji Island invasions. 

(McHugh, 1966, p. 2-19) 

Wargaming and model use will continue to grow in 
importance as funding is reduced and the need for analysis 
tools increases. Wargame adaptability uniquely applies to the 
ever expanding and diverse third-world military scenarios, the 
continuous redrawing of global country boundaries, and unique 
combinations of our own forces. 

B. PURPOSE AND GOALS 

The principal purpose of this thesis is to provide 
information about wargaming to prospective users. The first 
component is wargaming in general. The goal is to provide the 
reader with limitations that need to be considered and hazards 
that must be avoided when using conclusions derived from 
wargame data. The thesis contains a description of wargame 
features and a discussion concerning wargame limitations and 
hazards. The second component is how to design a wargame 
experiment to support C3 hypotheses testing. The goal is to 
enable the reader to test C3 hypotheses using a wargame 
experiment by providing a guide. A general description of the 
steps in planning a general wargame experiment is provided, 
including detailed information on wargame software 
considerations. A synopsis followed by a critique of an 
actual C3 wargame experiment is furnished apply the principles 
supported in this thesis. 
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C. ORGANIZATION AND SCOPE 



This thesis is an implementation guide. Chapter I 
provides the reader with a background on why wargaming is 
advantageous for research and will be even more useful in the 
future. It also includes the thesis purpose, goals, and 
organization. Chapter II goes into more detail on C3 wargame 
applications, types, features and basic requirements. Chapter 
III describes the limitations inherent to wargaming and 
hazards to avoid in wargame use. Chapter IV guides the reader 
through the steps for a general experiment design, including 
detailed information on the critical stage of software 
selection. Chapter V provides a synopsis of an actual C3 
experiment that used a wargame to evaluate a decision aid. 
Then the example is critiqued to apply the principles given in 
this thesis. The final chapter is a summary. 
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II. WARGAMING DESCRIPTION 



Before going into the specifics of wargaming 
considerations and experiment design, this chapter ensures 
that the reader understands the author's idea of what a 
wargame is, some general and C3 wargame applications, the 
types of wargames, and common features and basic requirements 
of a wargame. 



A. DEFINITIONS 

In order to begin with a common understanding of terms 
used in this thesis, the following definitions apply. 



• Models are simplified approximations of reality. A model 
is a representation of an object or structure, or an 
explanation or description of a system, a process, or a 
series of related events (McHugh, 1966, p. 1-4). The 
amount of detail and variables in a model can vary to fit 
its use. Some models look like what they represent, some 
are similar and some are symbolic. 

• A wargame is a theoretical representation, in accordance 
with predetermined rules, data, and procedures, of 
selected aspects of a conflict situation or military 
operation. A gaming exercise employs human beings acting 
as themselves or playing simulated roles in an environment 
which is either actual or simulated (PACER Systems, Inc, 
1975, p. 11). 

• A simulation is an operating representation of selected 
features of real-world or hypothetical events and 
processes (McHugh, 1966, p. 1-1). All-machine simulations 
(no human decision making) are not defined as games. All 
games however, are simulations. 
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There is a distinct difference between a simulation and a 

wargame that needs to be understood for C2 applications. 

In simulations, explicit rules and procedures within the 
simulation help define available choices and decisions. 
Conversely, in war games, players interactively make the 
pivotal decisions and choices. In this context, war games 
are simulations with participatory user interaction. 
(Airpower Research Institute, 1985, p. 3) 

Use of strictly simulation is not recommended for C2 

hypothesis testing experiments because human impacts and 

dimensions are not easily duplicated by a computer program. 

In model building, C3I is generally recognized as one of 
the most difficult areas to portray. Within the open 
simulation (man-in-the-loop) framework, some of the burden 
is removed from the model designer when the decision is 
made to allow the player to perform the command and 
control (C2) functions directly. The remaining processes , 
intelligence and communications, are somewhat more 
mechanical and, therefore, simpler to represent in a 
rational fashion. (National Defense University, 1986, 
Volume I, p. IX-1) 

In addition to understanding what wargaming is, it is also 
useful to understand what wargaming is not. Wargaming is not 
a technique for producing a rigorous, quantitative or logical 
dissection of a problem or for defining precise measures of 
effectiveness by which to compare alternative solutions 
(Perla, 1990, p. 164). Wargaming is not real. The level of 
abstraction depends on the model and the application. A 
wargame is not duplicable. The interplay of human decisions 
affects the simulated outcomes and therefore, does not allow 
the researcher to expect the same sequence of decisions and 
outcomes during a replay of a game. Wargame designers and 
analysts involved in research need to be careful to define the 
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results of a wargame in terms of why and issues raised, not 
what happened. A game should not be used to predict outcomes. 
They are best used to investigate processes. 

B. GENERAL WARGAME APPLICATIONS 

This section discusses why professional wargames are 
played in addition to its traditional use for training. Some 
of the added applications for wargames are to investigate not 
rigorously quantifiable problems, decompose complex problems, 
uncover new aspects of a problem, test operational plans, 
assist in the planning process, and focus on human behavior. 

1. Explore Not Rigorously Quantifiable Problems 

For this type problem, the choice of which variables 
form the basis of a problem and the relationships of the 
variables are not easily determined. An example of a not 
rigorously quantifiable problem is determining the United 
States strategic forces necessary to deter nuclear war. A 
rigorously quantifiable problem would be determining the 
impact point of a bomb dropped by an aircraft. Selecting an 
output measure, determining relevant variables, and defining 
the underlying model is much more difficult in solving a not 
rigorously quantifiable problem. Wargames represent the only 
formal decision aid available for this type of problem. 

2 . Decompose Complex Problems 

Another application is to use a wargame as a tool for 
decomposing complex problems. Proper game preparation forces 
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a consideration of the range of possibilities for any 
assumption. This step provides a natural method for 
subdividing a problem into smaller parts. With wargaming 
experiments, one assumption at a time can be manipulated to 
examine its impact on the problem. The decomposition of a 
complex problem into separate models, each of which can be 
evaluated and the results later combined, is an aid to 
exploring a complex problem. However, caution should be 
applied when making the final combination. 

3 . Uncover New Aspects 

Wargaming can be a form of brainstorming to generate 
aspects of a situation that are not readily apparent. By role 
playing in a realistic game, participants involved in a 
complex scenario can explore alternative policies, discover 
unexpected alternatives, and sometimes, anticipate outcomes 
that differ from those originally envisioned (Airpower 
Research Institute, 1985, p. 44). This approach can be used 
in research to generate and test hypothesis concerning human 
interactions . 

4. Test Operational Plans 

Another use of wargaming is testing operational plans. 
The experiment requires a realistic model in order to apply a 
real-world situation that requires planning a complex sequence 
of actions. Insights may be gained as to how the plan will 
work in the real world by playing out the planned actions. If 
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the initial design of a game incorporates well-known critical 
factors into its models and procedures, the play of the game 
and the questions and issues it raises can lead to the 
discovery of other factors whose importance may have been 
previously unsuspected or undervalued (Perla, 1990, p. 181). 
The reader is reminded that insights not predictions of 
possible outcomes will be generated. 

5. Assist in Planning Process 

Since a wargame can be used to evaluate a plan, it can 
also be used to integrate plan evaluation into the planning 
process. First a plan is gamed and evaluated, then any 
deficiencies can be corrected in the next iteration. The idea 
is to create a planning, testing, and replanning cycle. The 
more complex the situation and therefore the plan, the more 
useful and practical is the wargaming application. The design 
of the game (organizing) , and the play and subsequent analysis 
of the game (exploring) form a circular chain or feedback 
loop, in which the questions and issues arising from one play 
of the game can reshape or reorganize the game system itself 
to make it a more accurate representation of reality (Perla, 
1990, p. 180) . 

6 . Focus on Human Behavior 

Wargames allow the researcher to focus on human 
behavior, specifically on human decision making. The models 
usually offer various outcomes based on a probability 
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distribution. Thus, the "roll of the dice" provides a wide 
range of possible results or snapshots of reality with which 
the players must deal (Perla, 1990, p. 276). The value of 
wargaming for decision making is in qualitative assessments of 
why decisions are made. The advantage of a wargame over an 
exercise is that in an exercise there are limits on the range 
of physical parameters and processes and on the range of 
potential decision choices participants can choose from. 

C. C2 WARGAME APPLICATIONS 

In model building, C3 is recognized as one of the most 
difficult areas to represent. Because wargaming uses a man- 
in-the-loop framework, some of the complexity in model 
building is reduced by allowing the player to perform the 
command and control (C2) functions directly. The 

communications function is more mechanical and can be simpler 
to represent in the model. 

There are at least four aspects of command and control 
activities and processes that can be approximated in wargames 
because they are characteristics that can be represented 
realistically enough in the games to support meaningful 
analysis. These aspects are: 

• Command roles. Decision making functions and the nature 
and scope of the force actions that result from the 
decision making responsibility are defined by doctrine and 
convention so there should be very little discrepancy 
between command roles in reality and those for a game. 
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• Hypothesis formation. Any time that decisions must be 
made in the face of uncertainty and incomplete 
information, the decision maker must construct from 
available information hypotheses as to the nature of 
events, situations, etc., about which he has no knowledge 
and make decisions as if those hypotheses were true 
(Ketron, Inc., 1979, p. 26). This holds true for gaming 
if the decision makers are playing a similar game role as 
the role they hold in reality. 

• Filtering and interpretation of information. The way a 
commander identifies items relevant to a decision by 
filtering the constant stream of information against his 
hypothesis to arrive at a decision is realistic in a 
wargame if the actual information that the decision maker 
is filtering and interpreting is realistic in format, 
content, and order of arrival. 

• Information categories. Given that the information that 
is provided is tactically realistic, and that the roles 
and reasoning just discussed reflect those which would be 
encountered in the real world, the requests for 
information to support command decision making should also 
be representative of real world operations (Ketron, Inc., 
1979, p. 27) . 



These aspects focus on the exercise of decision making along 
the chain of command and are more dependent on the human 
element of C3. 



D. TYPES OF WARGAMES 

Wargames can be classified in many and multiple ways. The 
following gives a definition for classification by technique 
of rule application, wargame purpose, and geographic scope and 
decision making level. 
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1. Classified by Technique 

The two techniques of applying rules are rigid games 
and free games. A technique choice for knowledge about the 
opponent's forces is open or closed games. 

Rigid games are conducted by following detailed, 
nondeviating rules. The game rules usually provide for chance 
events; the role and the range of chance results are not left 
to the judgment of an umpire or controller (Hausrath, 1971, 
pg. 123). All possible outcomes are predetermined and rules 
provide for one possible outcome for each possible result of 
chance. For example, if there is a 50:50 chance of either of 
two outcomes, the rules determine how the result will be 
resolved such as a coin toss. Rigid games have extensive 
rules and are more complex to write for computer simulation. 
Rules provide for identical procedures, methods, and source 
data in each run and therefore are believed to provide more 
consistent and reliable data. 

A free game is conducted with fewer fixed rules and 
the assessments are made by an umpire or controller. The 
controller bases judgments on military experience. It is less 
tied to the mechanics of play and is therefore, less reliable 
than a rigid game and is usually more applicable to training 
use . 

The open game is simple and informal. The opposing 
players have full knowledge of the positions and actions of 
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their own and their enemy's forces so there is no need for an 
intelligence model. The open game is most often used for 
training . 

The closed game is more complex and formal. Each team 
has first hand knowledge only of their own forces and partial 
knowledge of the opposing team is received by an intelligence 
model. The closed game more closely approximates reality and 
is more useful in command and control hypothesis testing. 

2. Classified by Purpose 

When the primary purpose of a wargame is to provide 
the players with decision making experience, the game is 
educational. When a game is conducted in an attempt to obtain 
information and data that will help the responsible commander 
to make decisions, the game is an operational planning type. 
If the purpose is hypothesis testing, the game is an 
experiment for research. 

3. Classified by Level 

This categorization combines geographic scope and the 
level of decision making. The three levels are : 
global/strategic, theater /operational, and local/tactical . 

In global/strategic games, the participants are top 
level decision makers, and the goal is to improve their 
perspective, test strategies, and identify important issues at 
that level. The primary output of these high-level games is 
qualitative, consisting typically of game narratives with some 
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interpretations of events and little numerical data (Perla, 
1990, p. 170) . 

Commanders in chief of the unified or specified 
commands are the primary decision makers for the 

theater /operational level wargame. These games are designed 
to explore specific issues and identify strategic, 

operational, and tactical problems with the theater scope. 
The output is usually similar to that obtained for 
global /strategic games however, theater /operational games more 
often run the game more than once with the same players to 
generate more numerical data. 

The primary decision makers in the local/tactical 
category are usually battle-group commanders or below. These 
games focus attention on force levels and tactical 

deployments, weapon and sensor performance, and 

interrelationships among various warfare areas. The outputs 
are a combination of qualitative and quantitative with more 
numerical data than the other game categories. The number of 
times the game is run with the same players tends to be 
higher. 

E. COMMON FEATURES OF WARGAMES 

The following basic characteristics are present in all 
wargames . 

• Simulates a military operation. 
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• Is usually at least a two-sided representation of forces 
opposed to each other. 

• Is conducted in accordance with data, rules, and 
procedures that conform to accepted military doctrine. 

• Represents an actual or predicted real-life situation. 

• Provides for the employment and tactical movements of 
units in each force. 

• Uses some graphic means of identifying units, weapons, and 
positions of forces. 

• Provide a system for taking into account the firepower and 
other capabilities of the forces and equipment involved 
and a means for assessing the effects of combat. 

• Includes a clash of opposing forces in fire fights, 
battles, or prolonged campaigns. (Hausrath, 1971, p. 11) 
and (McHugh, 1966, p. 1-12) 



A wargame played for logistic requirement determination would 
have exceptions to the above list of characteristics. 



F. BASIC REQUIREMENTS FOR WARGAMES 

The following is a list of the basic requirements that are 
common to all wargame applications. These requirements need 
to be determined early and include: personnel; facilities and 
equipment; rules, data and procedures; and a scenario. 

1. Personnel 

Because wargaming involves humans in the decision 
process, an especially important requirement is knowledgeable 
personnel. For C3 experiments, it is critical that the 
players be highly qualified and be assigned to positions 
appropriate to their experience and training. The game 
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designers and analysts should be militarily knowledgeable and 

have the required mathematical, statistical, and related 

technological knowledge and skills. 

These scientists must translate military characteristics 
and actions into models and data that may be recorded and 
tabulated. They must then compute the interactions. They 
must assure that chance events and results are treated in 
a sound mathematical and statistical manner to represent 
realistic probabilities. In supervising the recording and 
processing of game data, and in the interpretation of 
results, proper allowance must be made for the effect of 
assumptions, approximations, and change results. All 
elements must be managed within the limits of reliability 
imposed by the original input data and data-processing 
methods. (Hausrath, 1971, p. 84) 

Most games currently in use are played on computers. Some 

members of the team of designers and analysts must be skilled 

in designing and programming, modifying, and troubleshooting 

the models processed in the computer; the supervision of the 

computer input preparation; and the interpretation of computer 

outputs . 



2. Facilities and Equipment 

The complexity of gaming facilities and equipment has 
grown with technology. In the 1800 's games were played with 
physical pieces representing equipment and units, now games 
are usually played on interacting computers with display 
monitors. Facilities usually consist of two or three 
adjoining rooms, each having access to a connecting corridor. 

3. Standard Rules, Data, and Procedures 

Standard rules, data, and procedures are provided in 
the form of a game manual or handbook. Also a set of 
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reference data such as the weapon capabilities and the 
quantities of each weapon available on each unit is required. 

4 . Scenario 

A statement of the situation is required for the 

initiating and conducting of any game. It may be a briefing 

and/or a detailed game directive. The statement of the 

situation must include all the "givens" for the game. 

These "givens" include a statement of the geographic 
locale, the forces and weapons available to each side, the 
time frame in which the action is set, and the political 
and other constraints applicable to the game. Also, 
"givens" customarily include a resume of the conditions 
leading to and resulting in the initiation of conflict, 
the starting deployment of forces involved, and, above 
all, orders or instructions to each commander stating the 
purpose and objectives of the game, his special mission, 
and related details. (Hausrath, 1971, p. 84) 

This chapter has defined the difference between a 
simulation and a wargame, wargame applications other than 
training and logistics, and some common features and basic 
requirements of all wargames . The next chapter addresses the 
limitations and hazards inherent in wargaming use and 
conclusion application. 
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III. WARGAMING CONSIDERATIONS 



Wargames and the conclusions derived from wargaming are 
often misunderstood, criticized, and even opposed. This 
circumstance has resulted from a misunderstanding concerning 
the nature of wargaming. When its limitations are understood 
and hazards are avoided, wargaming is a very valuable tool. 

A. LIMITATIONS 

It is critical for the wargame researcher and data user to 
fully comprehend the limitations inherent to any wargame. 
These handicaps can only be mitigated and not negated. 
However once their effects are understood, wargaming is very 
useful for valid experimental testing. 

1. Reality vs. Complexity 

Wargame designers tackle a built in conflict when 
trying to balance realism versus playability (detail versus 
simplicity) . The difficulty is in designing symmetry between 
the game system and the game player. If the game system 
becomes too complex when the designer attempts to accurately 
duplicate realism, the player's behavior may become unnatural. 
A complicated game system can artificially slow the players 
reactions to the game and make them more concerned with the 
mechanics of the game. However, over simplistic game systems 
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do not challenge the players and will not encourage decisions 

more like those made in a real situation. 

The key to realistic wargaming lies in balancing the 
player's experience in his decision making role with 
as accurate a representation as possible of the 
physical outcomes of his own decisions, his opponents 
decisions, and the objective dynamics of combat. 
Achieving this balance is difficult because realistic 
decision making requires giving the player realistic 
information and accurately representing the realistic 
effects of time. (Perla, 1990, p. 303) 

2. Artificialities 

The fog of war cannot be fully duplicated; realistic 
information is difficult to accurately represent in a wargame . 
Typically the players have too much information and of too 
high quality. Realistic provision for chaos, unanticipated 
successes and failures, "someone who doesn't get the word" and 
a commander not knowing what is happening elsewhere in the 
battle are difficult to represent in a wargame. 

Communication circuits as represented in a wargame 
usually perform too well. Message text and voice 
intelligibility are always good. There are no communication 
outages, and transmissions and delivery are much faster than 
in reality. 

Combat results can be modified by the game controllers 
to continue the game. An example would be to not allow all 
Blue carriers and flagships to be eliminated by the first Red 
strike, so Blue can continue to play. If this happens, the 
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combat results cannot be used to assess strategy and tactics 
used by either side. 

During the game play, the participants can become 
intensely involved but because the game is limited to work day 
hours with breaks for lunch, the combat stress is reduced. 
Also when the game breaks occur, the participants have an 
opportunity to review their situation. This assessment time 
would probably not be available in a similar real world 
situation . 

Each player approaches the game with a preconceived 
mental model of the world and will act based on their previous 
experience and knowledge. This is called framing. The 
information presented during the game is processed by this 
internal model. Because it contains a set of filters and 
sorting and weighing factors, not all the information is 
accepted. When a player makes a decision, the alternatives 
considered fit within their mental model. Therefore, the 
solutions for the player to choose from have been internally 
limited. 

Combat, fear, danger, losses and the unexpected are 
missing from a wargame. Morale, leadership, fatigue, and 
courage can be added to the list of factors that have an 
important effect on the outcome of a situation, but which 
cannot be predicted or measured. An example is whether a unit 
will stand and fight or break. The will to fight is not 
quantifiable or even understood. 
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Admiral Arleigh A. Burke put his finger on the central 
artificiality of wargaming when he said that "nobody can 
actually duplicate the strain that a commander is under in 
making a decision during combat." In a wargame, real 
forces do not deploy, real weapons do not engage, and real 
people do not die (Perla, 1990, p. 250). 

These elements are usually handled by assuming parity or 

equality for opposing forces. 

An additional artificiality is the assumed homogeneity 
of similar forces such as all pilots are equally skilled and 
all guns and missiles of the same type work equally 
effectively. These simplifications are necessary for game 
playability. 

The certainty that at some point the confrontation 
will escalate into warfare is a built in unnatural condition 
that affects the players' perception of the situation and 
their decisions. In a real world confrontation, the option of 
a diplomatic solution exists. As a result, the game 
commanders are less troubled by the possibility that their 
actions will precipitate a conflict that otherwise would have 
been avoidable, and tend to be more aggressive than they might 
be in a real world confrontation (Ketron, Inc., 1979, p. 23). 

3. Data Validation 

Data validation is a measure of how much confidence 
there is in the data for a given model. Many times for 
complex and difficult games, some of the real world data is 
not available. An example is a wargame concerning the 
benefits of using a new system in combat when it has not been 
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implemented or tested in the real world. Also the data can be 
unavailable because the system for the model has not yet been 
built. This is a "catch 22" predicament because a lack of 
real world data for complex and difficult situations can be 
precisely why they are wargamed. 

4. Workable Falsehoods 

To create a functioning wargame, the designer must 
apply some basic assumptions. In reality these assumptions 
are incorrect , but they are minor falsehoods that the designer 
and researcher can manage to deal with. 

a. The Right Variables Have Been Selected 

That the right variables have been selected is the 
first assumption. A close examination of the hypothesis and 
issues will identify a large number of variables. A good 

workable model will not be able to address all of the 
variables. It will only be able to deal with the set of 
variables that are high priority and only a few if any of the 
remaining variables. 

b. A ” Matched Set " Relationship Exists Between the 

Algorithm and the Data Base 

The algorithm for processing data can only work as 

well as the quality of the data and vice versa. 

For best matching, both components should be at a 
comparable level of acceptable perf ection . . . . An upper 
bound than on the fidelity of the algorithm would that it 
track the variables to no greater degree of granularity 
than that expected in the data base. (Patton, 1990, pp. 
43-44) 
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c. The Study of War Can Approximate an Exact Science 
Technology has advanced the art of wargaming and 

reduced in degree many limitations. There is one limitation, 
however that technology cannot impact. A person is unable to 
predict how they or any other individual will react in 
stressful and dangerous situations of war. This inability to 
predict human action and reaction is a major problem and a 
fundamental reason why the social sciences are generally less 
exact than physical sciences (Hausrath, 1971, p. 276). 

d. The Doctrine for Tactics and Data Used for the 
Wargame Design is Current 

The rapid advances of technology which produces 
new or improved weapons and systems contributes to doctrine 
evolution and input data change. Continuous revision of 
doctrine and data used in a wargame ' s design is not practical. 
Between periodic updates to reflect revised doctrine and data, 
a wargame will be out of date. 

e. A Wargame is Repeatable 

In every human-decision game, the players learn 
from experience. Therefore with each additional run of the 
game, the players have knowledge that they did not have in the 
previous game and this will affect their decisions. It is 
still useful to repeat a wargame to confirm the consistency of 
results or the lack of consistency. In general the 
reliability, validity, and applicability of the results of 
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reliability, validity, and applicability of the results of 
human-decision games are more a matter of judgment than of 
measurement (Hausrath, 1971, p. 287). In addition, if a 
stochastic method of determining the outcome of chance events 
is used, the same decision can result in a different outcome. 

5. Cost 

The four massive requirements of data, time, staff and 
computer support add up to big costs. Although a wargame is 
less expensive than a fleet exercise, the preparation, conduct 
and evaluation time for a wargame should not be 
underestimated. Pregame planning and postgame analysis can 
require several months of work. Player training and playing 
time takes them away from their jobs. 

6. Stochastic vs. Deterministic 

A simplistic description of the issue is whether it is 
more important to obtain expected results (firing 200 
things each having a probability of kill of 0.5 will cause 
100 kills); or whether it is more important to allow for 
all possible outcomes (firing 200 things as above could 
cause 100 kills, or 200, or zero etc.). (Patton, 1990, p. 
44) 

Some designers believe that a game will be more 
realistic if chance is portrayed stochastically so the players 
will be forced to consider unexpected and unusual outcomes. 
The opposing argument is that for a game's very finite set of 
encounters an outcome from the tails of the distribution could 
easily skew the game results. 
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B. HAZARDS 



The following is a list of wargaming considerations that 
are hazardous to the accurate results of game play. The 
researcher must anticipate and be vigilant to avoid these 
conditions when selecting a game, when the game is played and 
when representing conclusions. 

1. Poor Documentation 

The history of modeling shows that poor documentation 
creates poor results. Modeling documentation is critical when 
determining appropriate applications. The ongoing process of 
wargame revision can make the initial documentation obsolete. 
In addition, personnel turnovers deplete the corporate 
knowledge base concerning details and assumptions in models 
used (Airpower Research Institute, 1985, p. 35). 

2. Uncooperative Players and Umpires 

A wargame is a competitive situation. The players 
bring their egos to the game and don't like to lose. They 
have pride and some will improvise. 

In order for the game to be successful, the players 
must play wholeheartedly and enter into the spirit of the 
game. There may be times in the game when a player is not 
busy and may become bored. Another tendency is for some 
players to "fight the problem" by seeing only the limitations 
of the game models and not the advantages. A weapon may not 
perform "correctly" or an outcome may not be what they expected. 
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Umpires sometimes have to apply subjective values 

judgments in determining an outcome and abuses can occur. 

Umpires can limit the information they give to players and 
"muddy the waters" with false or inaccurate data. Umpires 
can also be arbitrary and inconsistent in their handling 
of such factors. Similarly, there is a strong tendency 
for human umpires to lapse into reporting information in 
game terms rather than real terms. (Perla, 1990, p. 304) 

3. Misplaced Advocacy 

The results of wargaming are used to communicate and 
convince. The inherent ambiguity of a not rigorously 
quantifiable problem allows misinterpretation of wargame 
results. Because assumptions and data input can be 
manipulated, there is an opportunity to use the wargame to 
produce a desired outcome. The designer of a game has great 
power to inform or to manipulate (Perla, 1990, p. 182). 

This chapter listed and described the limitations that 
cannot be eliminated from wargaming to enable the reader to 
understand the restrictions within which a wargame attempts to 
approximate reality. It also provided a list of elements that 
are dangerous to useful game results. The next chapter 
contains a description of the steps in a general wargame 
experiment plan. 
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IV. GENERAL EXPERIMENT PLAN 



This chapter's aim is to provide a broad outline of and 
guidelines for designing a wargame experiment to test a C3 
hypothesis. The reader should be familiar with general 
experimental design and analysis of variance. A one or two- 
semester course in statistics would be sufficient. 

Experiment credibility hinges on the approach used to plan 
or design the experiment. A clearly defined approach helps to 
ensure that the final product is appropriate. The statistical 
design of an experiment is the process of planning the 
experiment so that appropriate data will be collected. The 
data is then analyzed by statistical methods resulting in 
valid and objective conclusions. There are two considerations 
to any experimental problem: the design of the experiment and 
the statistical analysis of the data. They are related 
because the method of analysis depends directly on the design. 
The goal is a cost effective experiment that supports informed 
decisions . 

The process for planning an experiment includes the 
following stages: conception, design, preparation, execution, 
analysis and report. A looping cycle of refinement is likely 
to occur when going from conception to design and from design 
to preparation. 



28 



The success of an experiment is measured by the degree to 
which the experiment results answer the requested issues and 
objectives, and the credibility of those results. A effective 
experiment requires a team of militarily knowledgeable C3 
personnel (User) and statisticians that are experienced in 
experimental design (Analyst) . 

Although all of the six stages will be described, the 
thesis is mainly concerned with the design and preparation 
stages . 

A. CONCEPTION 

Recognition of and a statement of the problem is the first 
step. This step includes going from the issues, to the 
objectives and then to the response variables. The importance 
of this stage cannot be overemphasized. The Users begin by 
writing a detailed statement of the issues to be investigated 
in the experiment. They next define a clear, unambiguous 
description of the specific objectives of the experiment and 
how they relate to the identified issues. Experiment 
alternatives should flow from and support the objectives. A 
clear statement of the assumptions is written. 

Relevant measures of effectiveness (MOE) and measures of 
performance (MOP) are determined and designed to provide the 
decision-maker with either a quantitative or qualitative 
assessment of the objectives. The environment and the threat 
must be chosen because they influence software selection. 
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Security considerations unique to the experiment are also 
determined during this stage. The Analyst can assist in 
helping to identify the experiment alternatives, assumptions 
and write the hypothesis. The Analyst makes sure that there 
is a clear linkage between the objective, alternative, and 
hypothesis . 

The Analyst will also assist the User in selecting the 
independent variables. These factors may be either 
quantitative or qualitative. How these factors are to be 
controlled and measured is also considered. The levels of the 
factors must also be chosen. The dependent/response variable, 
how it will be measured, and the probable accuracy of those 
measurements are also determined. The User and the Analyst 
must be careful to be certain that the response to be measured 
really provides information about the objective. 

B. DESIGN 

The focus at this step is to match the experiment 
objectives with available simulation models, analysis tools, 
data requirements, measures, hardware requirements, and 
security. The detailed experiment plan should have complete 
traceability back to the originating issues, objectives and 
requirements . 

A key aspect is determining whether the issues and 
objectives lead to a specific generic experiment category. 
Some generic experiment types include: demonstration, 
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optimization, comparison, evaluation, sensitivity and 
prediction. Deciding the experiment type can assist in 
writing the mathematical model for the experiment so that a 
statistical analysis of the data may be performed. Also 
included in this step is determining the order in which the 
data will be collected and the method of randomization. The 
number of replications to support the analysis is decided. 
The number of subjects (player), from what population they 
will be obtained, and to what condition(s) each player will be 
exposed are planned. 

This thesis concentrates on the experiment design steps 
that are unique to wargaming. Those steps are: designing the 
scenario, providing a data base, and setting up the software 
program. 

1. Scenario 

The game begins by placing the players in a situation 
that requires them to make decisions. The scenario or 
situation sets the scene for player decisions and provides for 
specific updates in the situation during the play. The 
updates can alter or influence the developing situation and 
can draw player responses to specific items of interest. By 
defining the setting and scope of player decisions, scenarios 
can direct the course of a game into either very narrow or 
fairly broad channels, depending on the game's goals (Perla, 
1990, pg. 204). The scenario can have a major influence on 
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player decisions so the User needs to be sure that the 
scenario allows the players enough decision-making flexibility 
to let the game meet the objectives. In order to focus on 
exploring the C3 factors and reasoning that affect specific 
types of decisions, scenarios should allow players as much 
freedom as possible in making decisions by minimizing 
artificial restrictions. 

The User needs to work closely with the scenario 
designer. The designer must understand the experiment 
objectives and how they are to be met in the game. By 
identifying the kinds of player activities and decision-making 
opportunities that are required to meet the game's objectives, 
the designer can ensure that those activities and 
opportunities can happen. The scenario writer structures the 
flow of game play to allow the User's needs to be met but 
without forcing the players to follow a rigid path. A 
critical node in the flow of the game, at which a player's 
decisions will lead inevitably down one or another of several 
major alternative paths (to order a battle force into a fjord 
or to hold it in the open ocean; to launch a strike 
immediately or await the enemy's attack first), may be called 
a decision point (Perla, 1990, pp. 208-209). These decision 
points need to have enough detail so that players will have a 
realistic range of alternatives, but the number of 
alternatives needs to be manageable for analysis. 
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A coherent scenario must be logically consistent and 
tie together all the elements of the experiment including the 
originating issues, objectives and requirements. Usually, it 
must also be a believable possible view of reality from the 
players perspective. The exception is when the objectives of 
the experiment are important enough that the players are 
willing to shelve disbelief to support exploration. 

The User should review the scenario designer's 
documentation including reasons behind choice of assumptions, 
factors included or excluded from the scenario, the use of 
particular sources of information, and any other decision. 

Control personnel, players and analysts each have 
their own perspective and requirements for information 
concerning the scenario. 

Control personnel need to understand the context of the 
game and the prerogatives and limitations under which they 
must operate. Players need the same information as 
control personnel, but it should be constrained to reflect 
their less-than-perf ect knowledge of their game world. 
Analysts need to know not only the full story of what and 
why, the "ground truth," but also the story as told to 
players and control, so that they may interpret the 
effects of information constraints. Finally, the future 
consumers of the game's issues must know not only the 
context of the game, but also how to distinguish scenario 
input from game-play output. (Perla, 1990, p. 211) 



The User also needs to plan how to distribute the 
scenario information to the game-control personnel, game 
players, game analysts and report readers. 
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2. Data Base 



In addition to understanding the situation, the 
players must have access to the game's data base. The data 
base includes all the information that the players would 
reasonably have access to in a real situation. The data base 
of the game contains information about the capabilities and 
number of forces, levels of logistics, and relative 
likelihoods of the occurrence and outcome of interactions 
between forces, and the physical or environmental conditions. 
It also is the doctrine, tactics and procedures for the use of 
the forces and equipments. 

The data base links the scenario and the mechanics of 
the game. It must provide all the inputs required to allow 
the game's models to reproduce the qualitative scenario 
conditions and to generate outcomes of interactions. 

The data base provided to the players should be 
tailored to their game role, to the types of decisions they 
should be making, and to the types of information they need to 
make those decisions (Perla, 1990, p. 212). Important data 
need to be easy to find or require a minimum of calculations. 
The players need enough information to be able to determine 
what possible outcomes will result from the various decisions 
they may make. 

Some data is easily quantifiable and available such as 
logistics input data, weapon and equipment performance that 
have been experimentally field tested, and information 
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available from records such as ship transit time. Some data 
will be unavailable such as for weapons never used in combat 
(tactical nuclear weapons). This data must be deducted or 
extrapolated from related data or assumed to be within 
reasonable limits. 

Qualitative data for judgments of unit discipline and 
morale, leadership of opposing commanders can be either 
assumed equal for opposing teams or can be assigned quantified 
values that represent different levels or degrees of these 
qualities . 

The C3 data base representation of doctrine, tactics 
and procedures is difficult to include. They change over time 
and sometimes are not well documented because they depend on 
the commander and his own command structure. The command and 
control data base should include the communication, EMCON, 
surveillance, EW and OPCON matrix. 

3. Software (Wargame) 

When selecting the wargame, the User and the Analyst 
must constantly evaluate how the experiment objectives 
determined during the conception stage will be applied in the 
game. This section lists and describes areas that the User 
and Analyst will examine in designing the game. These 
decisions form the game parameters. The software to use is 
the one that comes closest to these parameters. 
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a. Objectives 

Every wargame has objectives. The objectives 
determine the game's uses. For example, a game may be 
conducted to provide decision-making experience at one or more 
specified levels and types of command; another to provide 
information and data concerning the employment of specific 
forces or weapons systems, test an organization or 
distribution system, or evaluate a type of operation or a 
tactical doctrine (McHugh, 1966, p. 1-11). The game designer 
should have clearly identified how and in what ways the game 
could provide the type of experience needed to achieve its 
objectives. The User needs to match as closely as possible 
the experiment objectives with the game objectives. 

b. Scope and Level 

The selection of the scope and level for an 
experiment follow directly from the specific experiment 
objectives . 

War games may range all the way from a contest between two 
units of a single service to a simulated global conflict 
involving coalitions of nations, the efforts of all 
services, and the impact of conventional and nuclear 
weapons on military forces and civilian economies. Games 
may be tactical, strategical, or a mixture of both. Some 
emphasize air operations; others, land or sea operations. 
Geographically, games may embrace a limited area, a single 
area of operations, or several areas of operations. 
(McHugh, 1966, p. 1-13) 



Related to the scope of a game is the range of command levels. 
Command levels can range from individual unit commanders to 
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national leaders. The scope and range of command levels help 
to determine the basic military units that will be used in the 
game. If the lowest command level of naval forces is the 
squadron, then the smallest unit might be the division. 

c. Number of Sides 

Games can be one-, two-, and n- or multi-sided. 
A one-sided game is played under two or more different 
situations. Some examples ares to find the minimum time for 
the simulation of a replenishment operation when there is no 
opposition from the enemy; and when the players are opposed by 
the game controllers or a pre-programmed opponent. The most 
common wargame is the two-sided with teams of players 
representing each of the two major contending parties or 
coalitions (Blue vs Red) . A multi-sided game is less common 
and is most often used for diplomatic or political-military 
games . 

d. Amount of Intelligence 

An "open" game is when all players have access to 
complete information on each other's plans and forces. 
Advantages to the open game are a reduction in personnel, 
space and time requirements. A "closed" game is when the 
players have access only to intelligence that would be 
available in real-world conditions. The data generated in a 
closed game will be more reliable if the simulated flow 
information is incomplete or interrupted. 
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A closed game with information handling 
enhancements will make C3 experiments more credible. The best 
approach for a wargame to take in information uncertainty is 
to have more than one level of truth. Some information is 
progressively disclosed automatically as the game progresses, 
but more detailed and accurate information can be acquired by 
aggressive sensor use, additional information requests, and 
revealed by team members sharing their information. The fog 
of war can be simulated by controlling the treatment of 
information. 

Ideally the game should allow each player to make the 
following "mistakes": (1) Non-observation of entitles 
which are within detection range because of non-alerted 
operators, system failures, environment, and deception (or 
loss of track after initial detection); (2) Incorrect 
location of detected entities due to system bias, time 
late, lack of gridlock, etc. (multiple tracks); (3) 
Incorrect classification of detected entities due to 
noise, jamming, environment, deception, failure to 
correlate separate sources, delay, etc. (confusion of red, 
blue and neutral); (4) Failure to report events.... In 
addition to the errors of the above type, the gaming 
system should introduce mistakes of the "false target" 
type. These include radar, sonar and visual and ECM 
sightings. (Sovereign, 1983, pp. 12-13) 



All of the above can lead the decision maker to an incorrect 
assessment of the tactical situation. By allowing the 
mistakes to happen, C3 is more realistically represented. 
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e. Methods of Evaluation 



One method is called "free" umpiring and is 
dependent on the experience, the judgment, and the objectivity 
of the game controllers. An umpire can select from among a 
broad range of realistic outcomes the one that is not only 
probable, but that also furthers the purpose of the game 
(McHugh, 1966, p. 1-18). The advantage is a quick and simple 
method of evaluation. 

Rigid umpiring rules are based on data that 
reflect real-world interactions. Contacts, number of hits, 
and the amount of damage are determined by equations and 
computations performed manually or by computer. 

Semirigid evaluation combines both free and rigid 
techniques. Some situations are evaluated by qualified 
officers and others are determined by equations, 
f. How Time is Represented 

Time is a critical aspect of any wargame for two 



reasons . 

First, in reality, timing and speed of execution are often 
decisive in determining the success or failure of a 
military operation. Second, time management in a game 
very often determines the extent of activity that the game 
can explore. (Perla, 1966, p. 222) 



When a game progresses by predetermined and 
constant intervals, it is a time-step game. Force movement is 
conducted during the time interval and at the end of the 
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interval, interactions are evaluated and the status of forces 
is updated. For example, a game may define a single move to 
represent an entire day's operations. The shorter the time 
interval, the longer the time required for play. 

Another approach to time-increment moves is to 

vary the amounts of real time depending on the importance and 

intensity of activity expected during a given time span. 

For example, a pre-hostilities move may encompass ten to 
fifteen days of activity. The D-day move of the same game 
may represent just a single day. (Perla, 1990, p. 224) 

The time interval for various moves may be predetermined in 
game design or may be defined by the game controllers during 
the game. 

A critical event or event store game advances play 
from event to event. 

Another method is continuous time. Player 
decisions may be made at any time and the orders and force 
attempts to execute those orders are continuous. Continuous 
time is closer to reality and is more likely to encourage 
dynamic interactions that occur in real operations. The game 
may be played faster, at the same rate, or slower than real 
time. However, when the game- time to real-time ratio is not 
one-to-one, there is a potential for distortion in the 
planning process. When the game is speeded up, planning time 
is reduced (uses too much game time) and when the game is 
slowed, decision time pressure is unrealistically reduced. 
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g. Method for Treating Chance Events 

Since chance events occur in real war, a wargame 
must be designed to deal with the probability that particular 
conditions will occur and to the variety of possible effects 
that will result from a given set of actions. Two ships may 
be adversaries. Which ship will fire first? Which ship will 
get the first hit? Will the first hit inflict little damage, 
cripple or sink the ship? What will happen to the opposing 
ship in the meantime? This example of a simple action has a 
whole set of probabilities or chance outcomes. For many 
events, the probabilities have been determined from war, 
exercises and test data. When the data is not available, 
best-estimate values are made by qualified experts. Wargames 
must attempt to duplicate this complex of elements and factors 
that may occur in almost infinite variety in real war 
(Hausrath, 1971, p. 285). 

There are two types of game uncertainties: 
stochastic and deterministic. Stochastic uncertainties are 
often characterized as the "roll of the dice", have a 
probability distribution and use random numbers. 
Deterministic uncertainties have a known set of inputs which 
result in a unique set of outputs, and the probabilities of 
the occurrence is an average or expected value. Each model 
will usually require a combination of the two types of 
uncertainties and their correct application is critical. 
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Because many assessments are based on chance 

factors, the Analyst should carefully evaluate the reliability 

of results by first performing a sensitivity analysis. 

Wide deviations may occur within the data from which 
average or best-estimate values are derived. The ranges 
of variation of these values, within which the results 
will remain valid, i.e., within which the values can vary 
without changing the nature of the result, are determined 
by sensitivity analysis. .. .Sensitivity analysis does not 
correct the data or assumptions used; it does give the 
analyst some measure of the level of confidence that can 
or should be associated with derived findings or results. 
(Hausrath, 1971, p. 284) 



h. Models 

As defined in Chapter I, a model can be a 

description of a system, a process, or a series of related 

events . The macro model of a wargame is a complex 

configuration of micro models and an explanation or 

description of their interrelationships. The most common 

micro models for wargames are weapons, logistics, command and 

control, intelligence, communications, physical environment, 

and sensors. During game play, the models can be programmed 

to calculate decision results. 

If the scenario sets the stage for the game, and the data 
base provides the information required for players to make 
decisions, the game's mechanics allow those decisions to 
be implemented and inform the players about their effects. 
Wargame mechanics may be considered as two interrelated 
systems: models and procedures. The models translate data 
and decisions into game events. Procedures define in game 
terms what players can and cannot do and why, sequence the 
game events to allow for accurate recreation of cause and 
effect, and manage the flow of information to and from 
players and control. (Perla, 1990, p. 214) 
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All models should be documented so the User can understand the 
assumptions and algorithms contained within the model design. 

A definition of the C3 model's function is to 
plan, direct, coordinate, and control assigned and attached 
forces to accomplish a mission. Many C3 models accomplish 
this function by using a combination of humans controlling the 
higher echelon decisions and computer generated decisions at 
lower levels. Command and control processes require the use 
of the communication model and often act on information from 
the intelligence and sensor models. Command and control 
processes normally include orders to subordinates, queries to 
superiors and subordinates for information, and an information 
flow to update superiors and subordinates. 

Basic C3 models include the following entities 
with their attributes: 



• C2 units: location; type of combat unit representative of 
this headquarters unit; lower echelon units under the 
direction of this unit; description of the echelon that 
the headquarters represents; its nationality and its 
vulnerability to various kinds of attack 

• personnel: authorized quantity; on-hand quantity; fraction 
of the on-hand quantity available for duty; a factor 
representing a fatigue coefficient; and a capability 
factor 

• communications: its own organic communications facilities; 
access to communications facilities; or have a combination 
of both 

• facilities: size, vulnerability, mobility, level of 

capability (to determine response time) , and capacity 
(National Defense University, 1987, p. II-4). 
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These entities and their functions require the support of all 
the other models and need a direct or indirect interface, 
i . Validation and Verification Considerations 

Validation is a measure of how much confidence 
there is on the output of a given model. There are two 

approaches for validation. The first approach evaluates the 
input data, the basic principles of the model and its 
assumptions. The second method collects real world 

information and compares it with the output of the wargame. 
One of the most important duties of the User and Analyst team 
is to determine the validity of the game's results and 
processes . 

The validity of a game's results can best be thought of as 
the extent to which those results reflect reality as 
opposed to the artificiality of the gaming environment. 
This validity depends on the accuracy of the mathematical 
evaluations of operational capabilities used in the models 
of the game, and on how well the quantity and quality of 
information available to the players of the game reflect 
the levels of information likely to be available to their 
real-life counterparts. (Perla, 1990, p. 236) 

Assumptions and input data need to be carefully 
examined. The results can be no better than the validity of 
the data and the assumptions from which they are derived. 

The User also needs to understand the quality or 
utility of both the macro and micro models. This is 
verification: is the model working as advertised. In the 

course of game play, are the experiences and insights the 



44 



players may gain from it determined primarily by the game's 
scenarios, assumptions, and mathematical combat models, or by 
the decisions that the players are capable of making. When 
the game is dominated by its mechanics instead of the 
decisions choices, misconceptions and errors in understanding 
can result. Player decisions cannot be driven from scenario 
mandates and constraints. An example of a player decision 
choice constrained by the scenario is assigning a mission to 
deter war in a crisis but mandating all the answers to the 
strategic questions. The key to creating accurate models and 
systems, then, lies in the ability of the design to 
incorporate as many of the elements important to actual 
decision makers as possible while simultaneously minimizing 
the number and effect of extraneous factors (Perla, 1990, p. 
242) . 

In addition to reviewing the model developer's 
test plan and results, the users manual, the algorithm 
document including where they came from or how they were 
derived, the User can run tests and verify that the overall 
game model will work for their experiment. 

4. Support System Capability Requirements 

The capabilities of a wargaming support system should 
be considered. The system should have a wide range of 
capabilities including: 

• Multiple simultaneous users 
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• Multiple interactive consoles 

• File management for large data bases 

• Security for file access. 

• Graphic displays 

• Recording and replay 

This list should be considered as a minimum and additions will 
be dictated by the experiment plan. 

C. PREPARATION 

1. Schedule 

A final schedule of experiment activities should not 
be established until all aspects of conception and design have 
been determined. Time must be planned for writing the 
scenario, gathering and in-putting the data, player training 
pretesting, running the experiment and reserving facilities 
and hardware. 

2. Players and Control Personnel 

The number of players on a team and their decision 
making levels need to be adequate to meet the experiment /game 
objectives. The player roles need to be consistent with the 
geographic and operational scope and scale of the game. Each 
player needs to have a well-defined operational role and must 
understand their function within the group. Players in the 
roles of non-U. S. or threat roles may need special training or 
expertise. They must not only understand the technical 
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capabilities of their foreign weapons but also the tactical 
and strategic doctrine of the opposition. 

Command levels above and below the team level may be 
assumed by the game controllers. At this stage, the players 
and control personnel are identified and matched to the 
functions to be performed to conduct the game. 

Player and control personnel training and 
indoctrination are performed at this time. 

3. Forms and Game Directives 

The special data-recording forms and work sheets must 
be designed for the experiment and duplicated. The game 
directives, scenario, order-of-battle information and other 
materials used as preliminary information handouts to players 
need to be prepared and duplicated. 

4. Playtest 

Playtesting is the best way to validate the entire 

game system by verifying that all the micro models mesh 

together to form a workable macro model. As the word implies, 

playtesting combines playing the game by thoroughly testing 

its functions and its ability to meet its objectives (Perla, 

1990, p. 242). To ensure that the game design is complete and 

that all data and procedures needed are available and correct, 

the Analyst thoroughly tries out the various alternatives and 

attempts to make the game fail. 

Especially important is the system's response to unusual, 
unexpected, or extreme decisions players may make. This 
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is especially important for system games, particularly 
those controlled by computer, in which the stabilizing 
influence of a human umpire is missing. The player that 
'cracks the code' of the game may play the game very 
successfully, but the educational value of his experience 
and the operational insights to be obtained from his 
decision making may be nonexistent because he is only 
taking advantage of a loophole that exists in the game 
system but would not exist in reality. (Perla, 1990, p. 
234) 

This testing process enables the User to correct problems with 
the system, procedures, or data before the experiment begins. 
Preplaying also gives the game controllers an opportunity to 
familiarize themselves with the game mechanics and their 
responsibilities . 

D. EXECUTION 

This is the actual game playing and data collection 
process. In a two-sided game, each team makes the decisions 
and perform actions appropriate to their mission. The 
movements, actions, and interactions are recorded by the 
computer. The game controllers have three functions. First 
is to monitor the actions of the teams and to enforce the 
rules of the game. Second is to evaluate the interactions 
following the methods and data of the game and the third 
function is to answer queries from the teams with the 
information and intelligence that are appropriate. The cycle 
for each team and controller interaction is: team decision - 
team action - controller monitor - controller evaluate - 
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controller inform. The game controllers need apply the rules 
in as uniform an experimental environment as possible for each 
game run. 

It is very useful to conclude the experiment by requesting 
an overall assessment of the game and for specific comments 
about particular issues or aspects of play from the team 
members. 

E. ANALYSIS 

Experiment data was generated in various formats, 
quantities and data rates. In accordance with the objectives 
of the game, the data generated in the play of the game will 
be sorted, compiled, and the results interpreted. A number of 
different analytical techniques may be used and were selected 
in the conception stage. In this step, the Analysts apply the 
chosen methods of data analysis to the experiment data. 

F . REPORT 

There are three common report formats of which the User 
may request any or all. The first report is a quick look. It 
is written as soon as possible after the experiment and 
summarizes the actual conduct of the experiment as compared to 
the planned procedures. This report also advises as to 
whether the information gained will be applicable to the 
issues and objectives of the experiment and any possible 
problems anticipated in performing the data analysis. 
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A preliminary report can be provided no later than two 
weeks after experiment completion. It supplies more detail 
concerning the information provided in the quick look report. 
Benefits of the preliminary report include a preview of the 
format and completeness of the final report so the User can 
provide an opinion. 

The final report includes an assessment of whether or not 
the objectives of the experiment were accomplished and the 
conclusions. It can include an appraisal of the credibility 
of the experiment, recommendations for follow-on analyses of 
the data and follow-on experiments to more fully examine the 
experiment and/or spin-off issues, and lessons learned 
concerning the experimental processes, resources used, and 
design and analysis techniques. 

This chapter described the stages of the process for 
planning an experiment using wargaming. It concentrated on 
the experiment design steps that are unique to wargaming: 
designing the scenario, providing the data base, and setting 
up the software program. It is followed by an example of a C3 
experiment which was planned to evaluate a decision aid. A 
description of this experiment provides an opportunity for the 
reader to understand the application of the principles given 
in this chapter. 
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V. EXAMPLE EXPERIMENT 



This chapter's intent is first to provide the reader with 
an example of an actual C3 experiment by summarizing the 
report into the following stages: conception, design, 
preparation/execution, analysis and report. Then a list of 
evaluation questions and answers is given for appraising the 
example experiment plan. Although very little information 
concerning the wargame is provided in the report, a list of 
questions for a wargame critique is also provided. 

This C3 experiment investigates the effects of introducing 
a decision aid into a headquarters, and was conducted for the 
Defense Communications Agency (DCA) and the Naval Ocean 
Systems Center (NOSC) by ALPHATECH, Inc. The objective of the 
experiment was to learn how the Naval-Based Replanning System 
(N-KRS), a decision aid for planning Naval air strikes from 
aircraft carriers against land targets, can best support the 
strike planning process. The experiment used the tactical 
naval warfare simulation designated Research, Evaluation, and 
System Analysis (RESA) and which is located at the wargaming 
facility at NOSC. 

The Headquarters Effectiveness Assessment Tool (HEAT) was 
applied in this experiment to validate defensible conclusions. 
The HEAT measurement tool consists of a set of variables 
(effectiveness measures) and a rigorous set of methods and 
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procedures for applying the measures to experiments and for 
analyzing the results. HEAT measures fall into three broad 
categories; process, performance, and effectiveness. The 
measures are based on a theory of headquarters decision making 
operating in the following cycle: gathering information, 
assessing the situation, generating decision options, 
predicting the outcomes of each option, selecting a response, 
executing the response, monitoring its outcome and then 
repeating. The HEAT methodology was originally developed for 
DCA eight years ago to understand the headquarters planning 
process. The experiment described here extended the use of 
HEAT to examine the decision making issues associated with the 
introduction of a decision aid to support planning in a 
headquarters environment (ALPHATECH, Inc., 1989, pp. 1-2). 
This chapter is a summary of the experimental report prepared 
by ALPHATECH, Inc. The information presented is typical of 
the data available after an experiment. 

A. CONCEPTION 

Recognition of and a statement of the problem is the first 
step. A detailed statement of the issues to be investigated 
in the experiment is written. 

1. Navy Air Strike Planning Problem 

The purpose of Navy air strike planning is to achieve 
a mission goal while satisfying a number of resource 
constraints. Tradeoffs exist between weapon effectiveness, 
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weapon availability, aircraft capability, aircraft 

availability, fuel requirements, and other factors. The air 

strike planner may receive guidance from higher authority on 

the target priorities. In the experiment, the players were 

given a desired probability of a kill (Pk) for the targets to 

indicate the priority of the target, with a higher Pk 

indicating a higher priority. This is an example of an aspect 

of the experiment design which did not follow reality. 

A typical planning process (as observed in the experiment) 
starts by considering the nature of the target and the 
desired Pk, and calculating the bombs required to achieve 
that Pk ( weaponeer ing ) . The aircraft needed/available to 
transport those bombs is then considered, and fuel 
requirements for the bombing mission are calculated. The 
aircraft needed for SAM suppression, fighter escort, and 
jamming missions are considered, and total fuel 
requirements are calculated. If the total fuel needed 
exceeds the fuel available, then the planning process 
recycles, with changes being made until the plan is 
satisfactory. Planning is not necessarily done in this 
sequence, however. The planning process can start with 
any aspect of the strike as long as all of the relevant 
factors are eventually considered. (ALPHATECH, Inc., 1989, 
pp. 4-5). 



A number of factors in the air strike planning process 
were not included in the experiment. Examples are weather, 
navigation overflight restrictions and launch sequencing. 

2. Navy Knowledge-based Replanning System (N-KRS) 

The N-KRS decision aid was produced by NOSC 
specifically for Naval air strikes and was based on the KRS 
strike planning decision aid developed by the Air Force. The 
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decision aid checks for consistency in air strike plans and 

tracks resource availability and resource use. 

N-KRS allows the planner to specify the goals of an air 
strike and then suggests a resource allocation to meet 
those goals within the constraints of the resources 
available. The user interface is in the form of a 
"template" with 9 primary "slots." The values assigned to 
these slots constitute the strike plan. The user may 
specify values for any number of the 9 slots. N-KRS can 
function in a fully automated mode, in which the user 
specifies only the target for the strike and N-KRS fills 
in all of the remaining slot values based on available 
resources and a knowledge base of planning rules. It can 
also function in a fully interactive mode, in which the 
user specifies values for all 9 slots, and the decision 
aid merely verifies that the plan is consistent and does 
not violate resource constraints. (ALPHATECH, Inc., 1989, 
p. 9) 



N-KRS recognizes a number of different mission types. 
The relevant types for the experiment were Power Projection 
Air (PPA) missions (bombing missions), SAM Suppression 
missions, Air Escort missions, electronic countermeasures 
(ECM) , and refueling. The following slots in the N-KRS user 
interface were used: target, Pk, ordnance, aircraft, AC 

number, carrier, unit (squadron), time of launch and time over 
target . 

3. Experiment Objectives 

The experiment had three primary objectives: 

1. Evaluate the value of the N-KRS decision aid for Naval 
air strike planning. Specific relevant questions are: 

(a) Does the N-KRS have a measurable effect on air strike 
outcomes? ; 
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(b) How does the N-KRS assist the air strike planner? - 
reduce workload, - reduce time, - reduce errors, - allow 
more option consideration, - allow more recent information 
incorporation . 

2. Demonstrate the N-KRS decision aid to experienced air 
strike planners and obtain feedback on potential 
applications and areas for improvement. 

3. Apply and improve the HEAT methodology for collecting 
observational data on a commander's decision making 
processes and linking process measures to outcome measures. 
Add to database of results within the HEAT framework. 
(ALPHATECH, Inc., 1989, p. 2) 



The objectives need to directly relate to the issues 
of the experiment. 



B. DESIGN 

1 . Overview 

The objective of the experiment was to compare air 
strike planning with and without the N-KRS to learn how the 
decision aid can best support air strike planning. In the 
experiment, four players prepared air strike plans with and 
without N-KRS, and their strike plans were simulated using the 
RESA wargame. Data on aircraft lost during the strike and 
damage to the targets were generated by RESA and provided to 
the subjects for use in planning subsequent strikes 
(ALPHATECH, Inc., 1989, p. 12). 

2 . Experiment Conditions 

There were two major factors in the experiment: (1) 

the presence or absence of the N-KRS decision aid. Each 
factor had two levels: the difficulty of the scenario - easy 



55 



or hard. The design was a two-by-two factorial with four 
conditions; each player participated in all four conditions. 
In order to separate learning effects over time from the 
effect of using the decision aid, two of the players began the 
experiment in the manual condition and two began by using the 
decision aid (ALPHATECH, Inc., 1989, p. 12). 

For the easy scenario, initial planning was based on 
the player's assessment of the resources needed for the 
mission. After the simulation was completed, the player 
planned a second wave and had to take into account battle 
damage assessment and any losses of aircraft due to an enemy 
counterattack before the second launch. The difficult 
scenario was the same; except in planning the second attack, 
the player had to consider that the enemy had moved a number 
of its aircraft away from the target base. Also, the enemy 
counterattack on the carrier group used more resources than in 
the easy scenario. Additionally, the players planned a third 
strike that required them to assess the aircraft available 
from the first strike (taking losses into account) and the 
battle damage from the second strike. The third strike was 
not played out in RESA. 

Players in the experiment planned an air strike 
against three enemy air bases, using the resources of three 
Navy aircraft carriers. The mission was to destroy the enemy 
aircraft parked on ramps at the bases and to destroy aviation 
fuel stores at each base. At the beginning of each session, 
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the players were given background briefing materials that 
listed all of the assets available for the strike, the enemy 
order of battle at each of the targeted bases (intel), and the 
desired probability of kill (Pk) for each target. Their 
assets included five types of aircraft and three types of 
weapons. The briefing materials instructed the subjects to 
plan the strike in two waves, with the first wave hitting two 
specified bases, and the second wave hitting the third base 
together with a possible restrike of the lesser damaged base 
from the first wave (ALPHATECH, Inc., 1989, p. 4). 

3 . Hypotheses 

The major independent variables are the presence of 
the decision aid and the difficulty of the planning task. The 
dependent variables were divided into process measures and 
outcome measures. The process measures are situation 
assessment accuracy, planning errors, information sought from 
intel, time to complete plan, workload, and outcome prediction 
accuracy. The outcome measure is the success of the air 
strike. The goal of the analysis was to understand the effect 
of the independent variables on outcomes by analyzing their 
effects on the processes underlying those outcomes (ALPHATECH, 
Inc., 1989, p. 19). 

The following hypotheses were tested in the 
experiment: 
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• The decision aid will reduce the time needed to plan or 
replan an air strike. 

• Commanders will take advantage of the reduction in the 
time needed to plan to seek more current information about 
their own or the enemy's situation. 

• In the more difficult planning condition, time-to-plan 
will be longer, and the commander's subjective workload 
assessment will be higher. 

• The decision aid will reduce the commander's subjective 
workload during planning. 

• Commanders will show more proactive information seeking 
under conditions of lower subjective workload. 

• Proactive information seeking will lead to more accurate 
enemy situation assessment. 

• More accurate enemy situation assessment will lead to more 
accurate evaluation of the expected effectiveness of the 
plan (option) selected. The accuracy of situation 
assessment and effectiveness evaluation will contribute to 
achievement of the strike goal. 

• The use of the decision aid will result in a plan with 
greater internal consistency (fewer errors made during 
planning) . 

• Fewer planning errors will result in better strike 
outcomes (achievement of strike goal). 

• The effect of the decision aid will be greater in the 
difficult planning condition. (ALPHATECH, Inc., 1989, pp. 
19-20. 



4 . Scenarios 

In the experiment, all of the scenarios involved a 
two-wave air strike against three enemy bases. The overall 
goal of the strikes was to neutralize the air threat at these 
bases. The mission was the same for all four sessions, and 
was provided to each player before each session in written 
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briefs. Three available Navy aircraft carriers were located 
approximately 600 miles from the enemy bases. The enemy bases 
had air defenses in all scenarios, and Blue usually lost some 
aircraft in each wave. The wargame did not support Blue 
maneuvering to avoid the enemy air defenses, but Blue could 
send a SAM suppression mission as a part of each strike. 

5. Amount of Intelligence 

Initial intelligence information provided the enemy 
order of battle at each of the targeted bases. Intelligence 
reports gave battle damage assessment information within 15 
minutes after each strike (another falsehood) . The subjects 
could request additional intelligence reports from "national 
sensors" and did get them. The HEAT observer noted the time 
the commander made the request and instructed the RESA 
controller to provide the information after a delay of 10-15 
minutes (another falsehood) . The requested reports provided 
additional battle damage assessments and estimates of enemy 
aircraft remaining at each base. The intelligence reports 
were based on ground truth but were not totally accurate. 

6. How Time is Represented 

Each experiment sessions lasted approximately three 
hours and twenty minutes. First, the player was given an 
opportunity to develop or change an initial plan at a time 
rate of 1-to-l. The strike was launched when the plan was 
completed, and RESA was accelerated to run at 6-to-l rate. 
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After the launch, the planes took about 30 minutes (clock 
time) to reach their targets. Battle damage and attrition 
information was available in 15 minutes. The player was given 
an opportunity to replan based on the new information, and 
RESA was slowed to run at 1-to-l. After the second launch, 
the game resumed a 6-to-l rate, and RESA played the air strike 
to produce a final set of outcome data. 



C . PREPARATION\EXECUTION 

1. Players and Control Personnel 

Four experienced air strike planners participated as 
players. Two of the players came from the carrier Carl Vinson 
and had been active in air strike planning. The other two 
players were experienced but not currently assigned to air 
strike planning duties. The following describes the personnel 
set for the decision aid condition followed by the non- 
decision aid condition. 

The Blue RESA operator and the Controller /enemy commander 
were RESA lab staff. The N-KRS operator was a NOSC staff 
member involved in developing the decision aid. As 
replanning occurred, a strike file was created off-line 
for entry into RESA (this allowed an interface between N- 
KRS and RESA to occur with a negligible time delay) ; the 
operator handling this function was provided by NOSC. The 
Blue player was positioned so that he could see the RESA 
displays and talk with both the RESA operator and the N- 
KRS operator. The HEAT observer was able to record the 
interactions that occurred and to see the RESA and N-KRS 
displays . 

In the non-decision aid condition, the player provided 
instructions for the creation of the off-line RESA strike 
file. In this condition, the player was assisted by an 
individual who did weaponeering calculations to compute 
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probabilities of kill, and kept tallies of resources 
available (ALPHATECH, Inc., 1989, p. 15). 



2. Data Collection 

Three types of data were collected during the 
experiment: 



• data collected by the HEAT observer recording the 
interaction of the strike commander with the N-KRS 
operator in the decision aid condition and the planning 
assistant in the manual condition; 

• questionnaires completed by the players; 

• and strike outcome data from the RESA simulation. 



Data collected by the HEAT observer included: 



• time that any information was received; 

• all actions taken to obtain additional information, 
including the time of the action, the type of information 
requested and received, and the time of arrival of the 
requested information; 

• all comments by the player about the certainty of incoming 
information; 

• for each piece of new information received, the time 
replanning was initiated (if ever) in order to adjust the 
plan for new information, and the time that replanning was 
completed; 

• the time and nature of all interactions regarding changes 
to be made to the plan between the player and the N-KRS 
operator or the planning assistant; 

• all options for the air strike plan discussed by the 
player; 

• all verbal instructions given by the player to the RESA 
interface operator; 
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• all comments by the player about the possible outcomes of 
any air strike planning options being considered; 

• all comments by the player about the plan's likelihood of 
success ; 

• all comments by the player about best/worst case scenarios 
for the plan's outcomes; 

• planning errors noted and corrected (or not corrected), 
including the time the error was noted, the type of error, 
and the time the correction was initiated and completed; 

• and all comments concerning the decision aid, including 
ease or difficulty of use, features asked for that were 
not provided, and errors or confusion on the players part 
about the capabilities of the decision aid (ALPHATECH, 
Inc. , 1989 , p. 17 ) . 



The players completed a set of questionnaires 
following the launch of each strike. The first was a workload 
questionnaire. The second was to assess situation assessment 
accuracy including estimates of the expected number of targets 
at each base (aircraft) and the confidence levels in those 
estimates. The third questionnaire assessed option evaluation 
accuracy and included estimates of the number or percentage of 
targets expected to be destroyed at each base by the strike, 
estimates of the number of Blue assets (planes) expected to be 
lost and confidence levels for both estimates. 

The data available from RESA was the outcome of each 
air strike and on the enemy and friendly aircraft lost during 
the strike. Data also included the movement of enemy aircraft 
during the difficult scenarios which was used to calculate the 
number of enemy aircraft remaining at the bases at the time of 
the strike. 
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D. ANALYSIS 



The following will serve as examples of the analysis. The 
report also included analysis of the accuracy of estimated 
strike effectiveness, planning errors, strike effectiveness 
and situation assessment accuracy. 

1 . Time Needed to Prepare Strike Plan 

One of the primary hypothesis is that the decision aid 

will save time for the air strike planner. The mean time 

taken to prepare strike plans (in minutes) with and without N- 

KRS for the easy and difficult experiment conditions were 

computed. The planning times did not include time spent 

adjusting available aircraft counts for aircraft lost in 

strikes or destroyed when carriers were attacked. An Analysis 

of Variance (ANOVA) was also determined for the planning time. 

Overall, N-KRS did reduce the time needed to produce a 
strike plan; in the Analysis of Variance (ANOVA) for 
planning time, the main effect of the decision aid was 
significant at the .10 level. However, the effect of the 
decision aid on planning time was concentrated in the 
first wave of the strike. There was a significant 
difference in the time taken to plan the first wave of the 
strike with and without the decision aid (p<.10). In 
contrast, the time taken to plan the second wave of the 
strike was similar with and without the decision aid in 
both the easy and difficult conditions. (ALPHATECH, Inc., 
1989, p. 21) 



The analysis included studying whether the players 
allocated their planning time to different activities when 
using N-KRS. The HEAT decision making process provided a 
framework for analyzing the planning and decision making 
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activities affected by using N-KRS . In the experiment, the 
functions most relevant to air strike planning are monitoring, 
situation assessment, option generation and evaluation. For 
the monitoring function, the analysts studied the frequency of 
requests for additional intelligence information. All of the 
interactions between the player and the N-KRS operator or the 
planning assistant were recorded by the HEAT observer. For 
the functions of situation assessment and option generation, 
the recorded interactions were grouped into four decision 
making activities. These activities were: enemy situation 

assessment, option exploration, option evaluation, and 
commands (option selection) . A table showing the distribution 
was generated to show the mean percentages (the percentages of 
interactions falling into each of the four categories for each 
experiment session and these percentages were averaged across 
subjects and across the easy/dif f icult conditions). An 
example of the results was the total number of interactions 
observed in the manual planning condition was much greater 
(611) than in the decision aid condition (360), indicating 
that more time was spent planning in the manual condition. 

2. Accuracy of Enemy Situation Assessment 

Before each wave of the strike was launched, the 
players were asked to estimate how many aircraft they believed 
would be at the enemy bases when the strike occurred. 

The accuracy of enemy situation assessment was measured by 

the absolute value of deviation between the estimates 
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given by the subjects and the ground truth about the 
number of enemy aircraft at each base, where a smaller 
deviation indicates a more accurate estimate. .. .The mean 
deviation was slightly smaller without the decision aid (a 
mean error of 38.4 aircraft without the aid versus 51.8 
aircraft with the aid) and the difference is not 
statistically significant. The only significant 
difference in enemy situation assessment accuracy was 
between the easy and difficult conditions. (ALPHATECH, 
Inc . , 1989 , p. 29 ) 



E . REPORT 

1. Organization 

The experiment report contained six chapters titled: 
Introduction, Navy Air Strike Planning, The N-KRS Decision 
Aid, Experiment Concept and Design, Experiment Results, 
Implications for Improving N-KRS and HEAT and Conclusions. It 
contained charts, figures, tables and diagrams. The appendix 
included subject briefing materials, questionnaires and data 
collection forms and a correlation matrix. The Experiment 
Concept and Design chapter was further organized into the 
following subchapters: 



• Experiment Overview; 

• Experiment Conditions; 

• Scenarios; 

• Intel Information; 

• Experiment Set-up and Personnel; 

• Subjects; 

• and Data Collection. 
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2 . Results 



The report overview of the results is summarized as 



follows : 

The experiment results suggest that the strike planners in 
the experiment were able to develop equally effective 
plans with and without the N-KRS aid. The major advantage 
of the aid was that it saved time and prevented 
errors.... As expected, the use of the N-KRS decision aid 
was negatively related to planning time, subjective 
workload, and planning time, subjective workload, and 
planning errors. Contrary to expectation, it had little 
effect on information seeking, and little effect on the 
accuracy of planner's assessment of the enemy 
situation. . . .Few of the process measures in the experiment 
had any relationship to the overall effectiveness of the 
strike, as measured by the number of enemy aircraft 
destroyed. (ALPHATECH, Inc., 1989, pp. 37-38) 



The report provided a figure which summarized the correlations 
that were found between the variables in the experiment. The 
figure used solid lines to indicate Pearson correlations with 
a significance level (two-sided) of at least p<.15. Dotted 
lines were used to show expected relationships that the data 
did not support. A full correlation matrix was available in 
the report appendix. 

3. Conclusions 

The following paragraph contains excerpts from the 

report's summary of conclusions. 

Subjects in the experiment were able to prepare equally 
feasible and effective air strike plans with and without 
the N-KRS decision aid, but they prepared these plans more 
quickly and with fewer errors when N-KRS was used.... Use 
of the N-KRS aid altered the process by which the subjects 
produced their strike plans.... In interpreting these 
results, it is important to note that the planners in the 



66 



experiment were probably not yet as thoroughly familiar 
with the N-KRS as they would be after weeks or months of 
experience in using it .... Sub jects ' estimates of the 
expected effectiveness of the strike were less accurate 
when they used N-KRS .... Sub jects in the experiment often 
commented that they would have liked N-KRS to show then in 
a compact, summary format what weapons and aircraft were 
available for the entire task force. (A1PHATECH, 1989, 
pp. 45-46) 



F. CRITIQUE 

There must be a clear connection between the objective, 
alternative, and hypotheses. The experiment design and 
analytical plan continue this link. Finally, the conclusions 
should be about the alternatives and be based on the positive 
or negative analysis results concerning the hypothesis. The 
experiment report needs to make all these linkages apparent. 
The experiment report prepared by ALPHATECH, Inc. is a good 
example. The following questions can be used effectively to 
evaluate C3 experiments. The author has included a response 
pertaining to the N-KRS experiment. 

The objectives need to be clear and relevant, the 
assumptions reasonable, and the hypotheses germane. 



• Was a decision identified which requires answering now? 
Yes. The N-KRS is a working decision aid. 

• Was the real issue directly addressed? Yes. The real 
issue was how the N-KRS could best support the strike 
planning process. 

• What question was the analysis designed to answer? The 
primary question was: Does the N-KRS have a measurable 
effect on the air strike outcomes? The secondary question 
was: How does the N-KRS assist the air strike planner? 
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• Could the issue be settled by an answer to the question? 
Yes if both questions are answered because the second 
question concerns the process. 



The experimental design should find a balance between being 
comprehensive and executable. The measures need to be 
discriminatory and quantifiable; and the data replicable. 



• Were all major factors effecting the question included? 
Yes, the factors were the presence or absence of the N-KRS 
decision aid. 

• Were sufficient levels of the factors identified? Yes, 
there were two levels for the difficulty of the scenario: 
easy and hard. 

• Do the scenarios match the issue and the question? Yes. 

• Is the scope of the system being considered adequate? Yes. 

• Were forces and weapons correctly included? Yes. 

• Does the operational concept reflect real doctrine and 
tactics? Yes, except for using the Pk factor in strike 
planning to indicate target priority and the intel 
availability. 

• Are standard operating procedures followed? The report 
did not contain this information. 

• Was the experiment adequately trained and staffed? The 
background and experience level was appropriate for the 
participants. The report only says that player training 
in the use of the decision aid was limited. 

• Were the supporting models matched in level with the 
issue? No information given. 

• Were supporting models credible? The only model described 
in the report was the intel model and it was not credible. 

• Were high level measures used (MOFE and MOE rather than 
MOP)? No. 

• Do the measures succinctly answer the question? Yes. 
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• Were both performance and vulnerability considered? 
Vulnerability was not considered. 

• Was a trial run performed? Information not given. 

• Were extraneous factors like learning controlled? Yes, 
two players began the experiment in the manual condition 
and two began by using the decision aid. 

• Were instruments adequate to measure variables? Yes and 
a variety of data was collected. 

The analysis must be correct and unbiased. The 
conclusions supportable. 

• Was the significance of each factor established? Yes. 

• Did the analysis use standard procedures such as ANOVA? 
Yes. 

• Were the conclusions based on the results? Yes. 

The experiment report does not provide information 
concerning RESA, the wargame. The report does not contain any 
information on the game data base, the method of gaming 
evaluation, the method for treating chance events and RESA 
validation considerations. 

In addition to evaluating the experiment plan and 
execution, the wargame itself needs to be examined. 
Appropriate questions for wargame analysis include the 
following list drawn from the appendix of The Art of Wargaming 
by Perla. 

These questions concern player preparation for 
participation in the wargame. 
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• What information was provided to participants prior to 
their arrival? 

• How are game objectives defined in preliminary briefings? 

• What information is briefed to participants before play 
begins? 

• How and to what level of detail is the scenario described? 
What is it? 



It is important to have a widespread view and 
understanding of the structure and style of the wargame. 



• What is the overall game structure and style? 

• Who are the players? Is there a team structure? From 
what commands to team members and leaders come? What are 
the names and real-world jobs of the principal players? 
How many sides are there in the game (one, two, or many)? 
What are the decision levels of the players, and how do 
they communicate? What are the responsibilities and 
limitations of the players, and how do these correspond to 
their roles? 

• What are the roles of control? How are command levels 

above and below the players represented? How do players 
and controllers/umpires communicate? What are 

controller/umpire responsibilities, powers, limitations? 

• What is the formal analysis plan? 

Knowledge concerning the game execution is important in 
critiquing a wargame. 



• What data and displays are available to the players? 

• During the course of play, what decisions are made by the 
players, and which are left up to others (control, 
umpires)? Are players' questions focused on what they 
should do, what they can do, what they must do, what they 
will do, or how can they do? 

• How are game events defined? What do control and the 
umpires not tell the players? 
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• How are events sequenced? What defines an activity? How 
is game time controlled relative to real time? 

• How does battle-damage assessment ( BDA) or event 

resolution work? Who does BDA? What techniques, models, 
data do they use and how? 



As mentioned in Chapter III, the human element can be a 
benefit or weakness of the wargame. 



• What are player's feelings about their roles and ability 
to influence events? What do the players see as the good 
points and the problems of the process? What special 
insights and ideas did the players bring to the game, and 
how has the play of the game affected them? 

• What are the attitudes of the controllers and umpires? 
How do they feel about their role, and how well are they 
carrying it out? Is there any source of disagreement? 



This list is not comprehensive but will give the reader a 
foundation for forming questions unique to an experiment. 

This chapter summarizes a real C3 experiment that applied 
wargaming to test the hypotheses and uses the principles from 
the previous chapters to critique the example. 
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VI . SUMMARY 



Naval military professionals need to be able to 
understand not only what wargaming can do, but also what it 
cannot do. This thesis provides information about wargaming 
to users who will apply conclusions derived from wargaming 
data in decision making and users of wargaming as a research 
tool for hypotheses testing. Game users need to understand 
and be able to ask appropriate questions concerning the 
process of using a wargame as an experiment tool. 

The first chapter explains that wargame use in general is 
rising due its ability to assist in evaluating combat 
capabilities, system improvements and new technology that add 
to the increasing complexity of warfare. Wargaming can 
effectively be used for identifying requirements and 
designing, developing, acquiring, training for and using major 
C3 systems. Wargaming is particularly suited to exploring C3 
issues because the game includes humans in decision making. 
However, wargaming is not a perfect image of reality and is as 
complex as the issues explored with it. Game data users need 
to be aware of wargame limitations and hazardous wargame 
issues. The goals of this thesis were (1) to enable the 
reader to ask intelligent questions concerning a wargame 
experiment before applying conclusions generated from wargame 
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data, and (2) to enable anyone desiring to use a wargame as an 
experimental tool to have a guide for the experiment plan. 

The second chapter ensures that the reader understands 
that a wargame employs human beings as decision makers and a 
simulation uses programmed rules to generate machine 
decisions. A wargame is not real, not duplicable and not 
useful for predicting future outcomes. The best use of a game 
is to investigate processes. Some general wargame 
applications include: to explore not rigorously quantifiable 
problems, to decompose complex problems, to uncover new 
aspects of a problem, to test operational plans, to assist in 
the planning process and to focus on human behavior. 
Applications exploring aspects specific to C3 are command 
roles, decision hypothesis formation, filtering and 
interpretation of information and information categories used 
in making requests. Types of wargames are defined by: 
technique choice for applying rules (rigid or free) and 
opponent force knowledge (open or closed) , the purpose of the 
game (educational, operational planning or research), and 
geographic scope combined with the level of decision making 
(global/strategic, theater/operational or local /tactical ) . A 
list of common features unique to wargames includes: a 
wargame simulates an actual or predicted real-life military 
operation involving a clash of opposing forces; usually 
represents at least two forces and provides a means for their 
employment; follows accepted military rules and procedures; 
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and provides a means for identifying and applying the 
firepower of units and weapons. The basic requirements for 
any wargame are qualified personnel, appropriate facilities 
and equipment, a game manual which provides the rules, data 
and procedures to be applied during the game, and a scenario 
statement of the situation. 

To avoid a reader misapplication of wargame data, the 
nature of wargaming, its limitations and potential hazards 
were discussed. The conflict between balancing reality and 
complexity, and the artificialities such as too much high 
quality information provided by communication circuits that 
work too well, and data validation problems contribute to 
wargame limitations. To facilitate game playability, the 
following falsehoods are acceptable: assume that the right 
variables have been selected for the experiment, that a 
"matched set" relationship exists between the algorithm and 
data base for the game, that the study of war can approximate 
an exact science, that the game uses current tactics and 
doctrine, and that a wargame is repeatable. Wargaming 
considerations that are hazardous to the accurate results of 
game play are poor documentation, uncooperative human players 
and misplaced advocacy. 

Chapter IV described a broad outline of and guidelines for 
designing a wargame experiment. Experiment credibility hinges 
on the approach used to plan the experiment. A clearly 
defined approach is critical to ensure that the final product 
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is appropriate for the accurate analysis of the experiment 
results. The first step of the conception stage includes the 
clear statement of the problem issues, objectives and response 
variables. Identifying the experiment alternatives and 
assumptions, writing the hypothesis, and selecting the 
independent and dependent variables are also included in the 
first step. The design stage matches the experiment 
objectives with available simulation models, analysis tools, 
data requirements, measures, hardware requirements and 
security considerations. The detailed experiment plan should 
have complete traceability to the originating problem and 
issues. The design steps unique to wargaming are: composing 
the scenario to allow player decision flexibility, tailoring 
the data base to the game objectives and selecting the 
software. When selecting the game, the game objectives, scope 
and level, number of sides, amount of intelligence, methods of 
evaluation, representation of time, method of treating chance 
events, available models, game validation and verification, 
and support system requirements all need to be evaluated so 
the best possible matching of the game capabilities and 
experiment requirements occurs. Experiment preparation 
consists of establishing a schedule, selecting players and 
control personnel, creating forms and game directives and 
playtesting to validate the game system. During execution, 
the game is played and data collected. Analysis of the data 
precedes writing the report. 
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The last chapter applies the principles from the previous 
chapters to critique an example of an actual C3 experiment. 
It summarized the N-KRS experiment report into the stages of 
conception, design, preparation/execution, analysis and 
report. An appraisal of the example experiment design, based 
on the information contained in the report, was provided by 
listing evaluation questions and answers. Very little 
information about the wargame was given in the report so only 
questions for a wargame critique was provided. 

The author has provided a summary on wargaming and an 
implementation guide for experimental wargaming. Although 
wargaming is an art, the reader is now better prepared to 
identify and avoid circumstances of wargame abuse and data 
misinterpretation . 
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